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Abstract 



In this correspondence, we introduce a minimax regret criteria to the least squares problems with bounded data 

uncertainties and solve it using semi-definite programming. We investigate a robust minimax least squares approach 

that minimizes a worst case difference regret. The regret is defined as the difference between a squared data error 

and the smallest attainable squared data error of a least squares estimator We then propose a robust regularized least 

^^ , squares approach to the regularized least squares problem under data uncertainties by using a similar framework. We 

^0 , show that both unstructured and structured robust least squares problems and robust regularized least squares problem 

^ I can be put in certain semi-definite programming forms. Through several simulations, we demonstrate the merits of 

the proposed algorithms with respect to the the well-known alternatives in the literature. 



Index Terms 

Least squares, deterministic, regret, regularization, minimax. 
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I. Introduction 



We study estimation of a deterministic signal observed through an unknown deterministic data matrix under 
additive noise [1]. Although the observation matrix and the output vector are not exactly known, estimates for both 
the data matrix and the output vector, as well as uncertainty bounds on them, are given [2], [3]. When there are 
uncertainties in the model parameters, a common approach to estimate the input signal is to use the robust least 
squares (LS) method [1], since the classical LS estimators perform poorly when the perturbations on the model 
parameters are relatively high [l]-[3]. Although the robust LS methods are able to minimize the data error for the 
worst case perturbations, however, they usually provide unsatisfactory results on the average [2]. Therefore, in order 
to counterbalance the conservative nature of the robust LS methods [1], we introduce a novel robust LS approach 
that minimizes a worst case "regret" that is defined as the difference between the squared data error and the smallest 

Suleyman S. Kozat, Nargiz Kalantarova, and Mehmet A. Donmez (skozat, nkalantarova, medonmez@ku.edu.tr) are with the EE Department 
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attainable squared data error with an LS estimator. By this regret formulation, we seek a linear estimator whose 
performance is as close as possible to that of the optimal estimator for all possible perturbations on the data matrix 
and the output vector Our main goal in proposing the minimax regret formulation is to provide a trade-off between 
the robust LS methods tuned to the worst possible model parameters (under the uncertainty bounds) and the optimal 
LS estimator tuned to the underlying unknown model parameters. Furthermore, after studying the data estimation 
problems in the presence of bounded data uncertainties, we extend the regret formulation to the regularized LS 
problem, where the regret is defined as the difference between the cost of the regularized LS algorithm [3], [4], 
and the smallest attainable cost with a linear regularized LS estimator. Under these frameworks, we provide the 
solutions for the proposed regret based minimax LS and the regret based minimax regularized LS approaches 
in semi-definite programming (SDP) forms. We emphasize that SDP problems can be efficiently solved even for 
real-time applications [5]. 

A wide range of applications in the signal processing literature deal with LS problems [l]-[3], [6]. However, in 
different applications, the performance of the LS estimators may substantially degrade due to possible errors in the 
model parameters. One of the conventional approaches to find robust solutions to such estimation problems is the 
robust LS method [1], [3], in which the uncertainties in the data matrix and the output vector are incorporated in 
optimization via a minimax residual formulation. Another approach to compensate for data errors is the total least 
squares method [2], which may be undesirable since it may yield conservative results due to data de-regularization. 
Furthermore, in many linear regression problems, the data matrix has a known special structure, e.g., Toeplitz [1], 
[2]. The performance of the minimax methods usually improve when such a prior knowledge is integrated in the 
problem formulation [1], [2]. 

In order to mitigate the pessimistic nature of the worst case optimization methods, the minimax regret approaches 
have been introduced in the context of statistical signal processing literature [7]-[10]. However, we emphasize that 
the robust methods studied in this correspondence substantially differ from [1], [3], [7], [8], [10]. The cost functions 
studied here are different than [1], [3], where the regret terms are directly appended in the cost functions. Although 
in [7], [8], [10] a similar regret notion is used, the cost function as well as the constraints are substantially different 
in this correspondence. Furthermore, note that in [7], the uncertainty is in the statistics of the transmitted signal. On 
the other hand, in [8] and [10], the uncertainty is in the transmitted signal and the channel parameters, respectively, 
unlike in here. In this correspondence, the uncertainty is both on the data matrix and the output vector Furthermore, 
since the cost functions are different for our formulations, the solutions to the LS problems presented in this 
correspondence cannot be obtained from [7], [8], [10]. We emphasize that the proposed methods are formulated for 
given perturbation bounds and note that such bounds heavily depend on the estimation algorithms used to obtain 
the observation matrix and the data matrix. In this sense, different estimation algorithms can be readily incorporated 
in our framework with the corresponding uncertainty bounds [3]. 

Our main contributions are as follows. We first introduce a novel LS approach in which we seek to find the 
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transmitted data by minimizing the worst case difference between the data error of the LS estimator and the data 
error with the optimal LS estimator. With this method, we aim to provide a trade off between the performance 
of the robust LS methods and the tuned LS estimator (tuned to the unknown data matrix and the output vector). 
Then, we develop a minimax regret approach for the regularized LS problem. We demonstrate that the proposed 
robust methods can be cast as SDP problems. In our simulations, we observe that the proposed approaches yield 
better performance compared to the robust methods optimized with respect to the worst case data error [1], [3], and 
the tuned LS and the tuned regularized LS estimators (tuned to the estimated data matrix and the output vector), 
respectively. 

The organization of the correspondence is as follows. We initially provide an overview of the problem in Section 
II. In Section III, we introduce the unstructured and regularized LS approaches based on the regret formulation. 
We then continue to study the structured LS approach. The numerical examples are given in Section IV. Finally, 
the correspondence concludes with some remarks in Section V. 

II. System Overview 

We consider the problem of estimating a deterministic vector u G K" which is observed through a deterministic 
data matrix. However, instead of the actual coefficient matrix and the output vector, their estimates H e jj^^x" and 
y G M™ and uncertainty bounds on the estimates are provided, where m > n\ Our goal is to find a solution to the 
data estimation problem Hu « y, such that (H + AH)u = y + Ay for deterministic perturbations AH G M™^", 
Ay G M™, which are unknown but a bound on each of the perturbation is provided, i.e., ||AH|| < Sh and 
ll^yll < ^Y, where Sh,Sy > 0. 

Even under the uncertainties, the unknown vector u can be naively estimated by simply substituting the estimates 
H and y into the LS estimator [4]. For the LS estimator we have u = H^y, where H^ is the pseudo-inverse of 
H [11]. However, this approach does not yield acceptable results, when the errors in the estimates are relatively 
high [1], [7], [8]. A common approach to find a robust solution is to use a worst case residual optimization [1] as 

u = argmin max ||(H + AH)u — (y + Ay)|| , 

u ||AH||<(5,f,||Ay||<(5y, 

where u is found by minimizing the worst case error in the uncertainty region. However, the solution may be highly 
conservative, since the solution is found with respect to the worst possible coefficient matrix and output vector in 
the uncertainty regions [2], [7], [8]. In order to alleviate the conservative nature of the worst case residual approach 
as well as to preserve robustness, we propose a novel LS approach, which provides a trade off between performance 

'AH vectors are column vectors and denoted by boldface lowercase letters. Matrices are represented by boldface uppercase letters. For a 
vector u, ||u|| = Vu^u is the <?2-norm, where u-^ is the ordinary transpose. For a matrix H, |jH|| implies the Frobenius norm. For a vector 
u, diag(u) is a diagonal matrix constructed from the entries of u. For a square matrix H, Tr(H) is the trace. denotes a vector or matrix 
with all zero elements and the dimension can be understood from the context. The operator vcc(-) stacks the columns of a matrix of dimension 
rax n into a ran X 1 column vector, and the operator ® is the Kronecker product [11]. 
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and robustness [7], [8]. The regret for not using the optimal LS estimator is defined as the difference between the 
squared data error with an estimate of the input vector and the squared data error with the optimal LS estimator 

7^(AH, Ay) = ||(H + AH)u - (y + Ay)f - imn ||(H + AH)w - (y + Ay)f , 

w 

where ||AH|| < 5h, ||Ay|| < Sy, for some Sht^y > 0. In the next section, the proposed approaches to the 

LS problems are provided. First, the regret based unstructured LS method is introduced. Then, the unstructured 

regularized LS approach is presented in which the worst case regret is optimized, where the regret is defined as the 

difference between the cost function of the regularized LS algorithm [4] with an input vector and the cost function 

with the optimal regularized LS estimator. Finally, the structured LS approach is studied. 

III. Robust Least Squares Methods 

A. Unstructured Robust Least Squares 

In this section, we develop a novel unstructured LS estimator based on a certain minimax criteria. Let H G R™^", 
y e M™, m>n, and 5h,5y > 0. We find u that is the solution to the following optimization problem: 

min max { ||(H + AH)u - (y + Ay)f - imn ||(H + AH)w - (y + Ay)f }. (1) 

" ||AH||<5„,||Ay||<(5y w 

Note that by unconstraint minimization over w, we have g(H,y) = ||(I— HH )y|p = minw ||(H + AH)w— (y + Ay)||^ 
H = H + AH, y = y + Ay. If we use the first order Taylor series expansion [11] for ^(H, y), then 

5(H,y)-.g(H,y)+2Tr|v,g(H,y)|^^jj.^^[AH Ay]|+0(||[AH Ay]f). (2) 

Clearly, the effect of this approximation vanishes as || [AH Ay] || decreases, however, we observe through our simu- 
lations that even for relatively large perturbations a satisfactory performance is obtained. Since -— ^ — l^v ^^ ^ = 

of ii=M,y=y 

2(1 — HH^)y and based on the approximation (2), the regret in (1) can be written as 

7^(AH, Ay) « II (H + AH)u - (y + Ay) f - 5(H, y) - 2Tr| Vg(H, y) I J^^jj ^^^ [AH Ay] 

= ||(H + AH)u - (y + Ay)f - ^ - 2(vec(D)^vec(AH) + 2Ay^P^y + 2y^P^Ay) 

= ||(H + AH)u - (y + Ay)f - (^ + c^h + h^c + Ay^b + b^Ay), (3) 



9H 'H=H,y=y' H 



where D = -^i;^ — 1^. „ > , Ptt = (I — HH ) is the projection matrix of the space perpendicular to the 

A „,^^ \ . A , ,„, A ,„, . , A 



range space of H, ^ = /(H,y), h = vec(AH), c = vec(D) and b = 2PTTy. 

In the following theorem, we illustrate how the problem of minimization of the worst case regret (3) can be put 
in an SDP form. 
Theorem 1: Let H e R™><", y g M™, to > n, and 5h,5y > 0, then 

min max | ||(H + AH)u - (y + Ay)f - /(H, y) - 2Tr{ Vg(H, y)l^ „ ^ ^ [AH Ay] }1 

u ||AH||<(5„,||Ay||<(5y I tl=tl,y=y J 

(4) 
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is equivalent to solving the following SDP problem 



min7 subject to 

l + i-T-6 (Hu-y)'^ 5hc^ Syh^ 

(Hu-y) I 5h\J -5y^ 

Shc (5hU^ tI 

Syh -Syl 91 



>0, 



(5) 



-r„---r+N - 



where .g(H, y) = y' (I - HH )y, C = Y (I - HH+)y, r, 6* > 0, c = vec 



a5(H,y) 



A 



, OH H.H,y.yj'b-2PHy^ 

U is an m X mn matrix and defined as U = I ® u^. 
Proof: The proof is in the Appendix. D 

As the special case, the pseudo-inverse of H is H"*" = (H H)^^H , when H is full rank. For this case, we 
introduce the next lemma to explicitly calculate V/(H,y). 



A 



Lemma 1: Let H e M™^", m > n, be a full rank matrix, y e R" and define g(H,y) = y'^(I - HH+)y, then 



2 H(H^H)-iH^yy^H(H^H)-i +yy^H(H^H)-i 



9H 

Proof: Taking the partial derivative of /(H,y) with respect to H^ based on [11] yields 
a.9(H,y) 



(6) 



dn. 



T 



e,ef (H^H)-iH^ - 2U{U^Uy^U^e^eJ{li^Hy^ii^ + ll{ll^Hy^eje 



2e7(H^H)-iH^yy^e, - 2e[(H^H)-iH^yy^H(H^H)-iH^e, 



(7) 



By definition, the transpose of the term in (7) is the ij entry of the matrix 



yy^H(H^H) 1 H(H^H) iH^yy^H(H^H)-i 



, hence the result in (6) follows. D 



From [11], we have 



g.g(H,y) 
dy 



2(1 HH+)y - 2P^y, 



(8) 



where Ptt = (I — HH^). By using Lemma 1 and (8) in (2), we get 



P"y 



AH Ay 



(9) 



-(yy^H(H^H)-i +HH+yy^H(H^H)-i 

(y^AHv + y^HH+AHv) - y^P^Ay 

(y^Vh + y^H+Vh) - y^P^Ay 

= C + z^h + h^z + 2Ay^P^y + 2y^P^Ay, 

where C = y^(I - HH+)y, h = vec (aH^), v = H+y, and z = -2(y^V + y'^H(H'^H)-iH^V)^. Equation 
(9) follows since AHv — Vh and B is an m x mn matrix defined as V = I v"^. 

If H is not full rank, then the pseudo-inverse of H can be written as H^ = VS^U by using the singular value 
decomposition [11], where U is an m x n unitary matrix, S is an n x n diagonal matrix with nonnegative real numbers 
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on the diagonal, and V is an n x n unitary matrix. The nonzero diagonal entries of S = diag[cri , (72 ,..., cTr , 0, ..., 0] 
are known as the singular values of H. In order to calculate VgfH, y)L-v tt ~ , it is sufficient to calculate -— — . 
Then, the derivations follow the full rank case using 



-S+U^ + V— — U^ + VS+— — =— — S+U^ + VS"^— — S"^U^ + VS+ 



T 



where the partial derivatives of U, V, and S with respect to ij*"^ element of H are derived in [12]. 

B. Unstructured Robust Regularized Least Squares 

A wide range of applications in signal processing literature require solutions to regularized LS problems [4]. In 
[3], a worst case optimization approach is developed to solve the regularized LS problem. To reduce the conservative 
nature of [3], we next develop a regret based regularized LS approach when the model parameters are subject to 
uncertainties. The regret for not using the optimal regularized LS method is defined as the difference between the 
cost function of the regularized LS algorithm with an estimate of the input vector and the cost function of the 
regularized LS algorithm with the regularized LS estimator as 

7e(AH, Ay) = ||(H + AH)u - (y + Ay)f + ^i\\uf - min{||(H + AH)w - (y + Ay)f + ^i\\wf}, 

w 

where || AH|| < Sh, and || Ay|| < Sy for some 6h,Sy > 0, and /i > is a regularization parameter. We emphasize 

that there are different approaches to choose /i, however, for the focus of this correspondence, we assume that it is 

already set before the optimization. Hence, we solve the the regularized LS problem for any given value of /i > 0. 

Given H G M™^" and m > n, y e M" we have 

min ^ max (||(H + AH)u - (y + Ay)f +Ml|uf -min {||(H + AH)w - (y + Ay)f +A*||wf }1. 

" ||AH||<(5H,||Ay||<(5y. L w J 

(10) 
Since H = H + AH, y = y + Ay and inserting the optimal regularized LS solution in (10) yields 

min max { ||Hu - yf + A^||uf - y^(I + ^-iHH^)-iy}. 

u ||AH||<(5„,||Ay||<(5y 

Note that (I + /i^^HH ) > 0, hence it is invertible. Denoting /i(H,y) = y (I + /i^^HH )^^y, and using the 
first order Taylor series expansion [11] for g(H,y) yields 

MH,y) = MH,y) + 2Tr|vMH,y)|^^jj.^^[AH Ay] | + ol^ll [AH Ay] f V (11) 

The following lemma is introduced to calculate V/i(H,y)|£v tt - in (H)- 
Lemma 2: Let H e M™^", y e R" and define h{U,y) = y^(I + fi-^HH^y^y, then 

^^fe^ = -2^-1(1 + M-^HH^)-Vy^(I + M-^HH^)-^H. (12) 
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Proof: Taking the partial derivative of /i(H,y) with respect to Hjj based on [11] yields 
5MH,y) 



5H,;, 



= -2/xef (I + ^-iHH^)-Vy^(I + M"^HH^)-iHej-. 



(13) 



By definition, the term in (13) is the ij*" entry of the matrix —2^(1 + /iHH )^ yy (I + /iHH )^ H, hence the 

result (12) follows. D 

Also, from [11] we have — ^^^ — - = 2(1 + /i^^HH )^^y. Then, one can write the regret in (10) based on the 



dy 



expansion in (11) as 



7e(AH, Ay) « (Hu - y)^ (Hu - y) + ^u^ u - (^ + c^* h + h' c + b' Ay + Ay^ b), 

where C = y'^(I + /i"^HH^)-iy, h = vcc(AH), b = 2(1 + fr^HH^y^y and c = vcc( — ^-^ 

\ on 

next theorem, we show that the minimization of the worst case regret (14) can be put into an SDP form 



(14) 
In the 



Theorem 2: Let H e R™><", m>n,ye M", and Sh,Sy > 0, then 

min max {||(H + AH)u - (y + Ay)f + /i||uf - (^ + c^h + h^c + b^Ay + Ay'^b)} (15) 

" ||AH||<(5ff,||Ay||<(5y 

is equivalent to solving the following SDP problem 

min 7 subject to 

j + ^-T-e (Hu-y)^ u^ 6hc^ Syh^ 
(Hu-y) I ShU -Syi 



u 





^iI 








Shc 


ShV^ 





rl 





Syh 


^Syl 








ei 



>o, 



(16) 



where /i > is a regularization parameter, t,9 > 0, £_ — y (1 + /iHH ) y, h = voc(AH), b = 2(1 



A 



5MH,y) 



dH 



,-vT, 



/i ^HH ) ^y,c = vcc( -— ^ — 1 for /i(H, y) = y (I+/i ^HH ) ^y and U is an to xmn matrix constructed 



by u, where U = I u . 

Proof: The proof follows similar lines with the proof of Theorem 1, hence omitted here. D 



C. Structured Robust Least Squares 

In many engineering applications the data matrix has a special structure, e.g., Toeplitz, Hankel, Vandermonde, 
as well as the perturbations on them [1], [2]. Leveraging this prior knowledge may improve the performance of 
the regret based minimax LS approach [1], [2]. Therefore, in this section, we study a special case of the problem 
(1), where the associated perturbations for H and y are structured. The structure on the perturbations is defined 



as follows as: AH = X^iLi "^jHi and Ay = X^iLi f^iYi' where H^ e 



yj G K™, m > n, and p are known 



but ai,f3i E K, i = 1, . . . ,p, are unknown. However, the bounds on the norm of a 



ai 0.2 



and 
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/3^ 



/3i P2 ... Pp 



are provided as ||q:|| < 5h, ||/3|| < 5b, where Sht^b > 0. We emphasize that this 



mm max 

" \\a.\\<5H,\\l3\\<5E 



|H(a)u - y(/3)f - min ||H(a)w - y(/3)f 
w 



formulation can model a wide range of structural constraints. We seek to solve the following optimization problem 

(17) 

W J 

where H(a) = H + ELi «^H„ and y(/3) = y + Y.L1 ^^y^■ 

We point out that minw||H(Q;)w-y(/3)f = ||(I-H(Q;)H(a)+)y(;a)|[2, where H(a)+ = (H(Q;)^H(Q;))-iH(a)'' 
is the pseudo-inverse of H(q:) and Pq, = (I — H(q;)H(q:)+) is the projection matrix of the space perpendicular 
to the range space of W.{ol). Here, H and H(q;) are assumed to be full rank. We use the first order Taylor series 
expansion in order to express the term y(/3)-^(I — H(Q:)H(Q;) + )y(/3) as 

y(/3f(I-H(a)H(a)+)y(/3)=y^(I-HH+)y + 2Tr{V[^ /3]l|Piyinifa ^^^^W /3]} 

+ 0(||[a /3]||2). (18) 

If we denote the regret term as Tl{a.,j3) = ||H(q;)u — y(/3)|| — ||(I — H(Q;)H(Q;) + )y(/3)|p, then the regret can 
be written based on (18) as 

7^(a,/3)«||H(a)u-y(/3)f -y^(I-HH+)y-2Tr{V[^ /3]llPiy|l'll J, /3]^o[" ^l^' ^^9) 

We introduce the following lemma to compute the last term in (19). 

Lemma 3: Let H,Hi, . . . ,Hp e M"''", y G M", H(q;) =H + X;Li"»Hj, a= [ai, as, ... ,ap], where m > n, 

and let H and H(q;) be full rank. If we define g{H{a),y) = y* (I - B.{cx)ii{cy.) + )y, then 



-2y^P^H,H(a)+y. 

Proof: By using the result of Lemma 1, 

5.g(H(a),y) / 9g(H(a),y)^aH(a) | 

= ir 



9ai I 9H(q;) (Jofi I 

= Trj (" - 2(H(a)(H(a)^H(a))-iH(a)^yy^H(a)(H(a)^H(a))-i + yy^H(a)(H(a)^H(a))-i)') (^^) } 
= -2y^PiH,H(a)+y, 



A 



where Pa = H(q;)H(q;)+ is the projection matrix of the range space of 11(a). D 



d 
From Lemma 2, it follows that -r— 

da 



y(/3)^Piy(/3)] = -2y(/3)^PiH.H(a)+y(/3). Also, A [y(/3)^Piy(/3) 
2y(/3)^(I-H(a)H(a)+)y, = 2y(/3)^Piy,. If we denote ^ = y^(I-HH+)y, b = Va\\P^y{(3)r\a=0.f3=0^ 
and c = V/3||PQ,y(/3)|p|Q,^Q [j^q, then based on (18) we obtain 

y(^)'^(I - H(a)H(a) + )y(;a) « ^ + h^a + a^b + /3^c + c^/3. (20) 

By using the result (20), in the next theorem we show that the problem of estimating u by minimizing the worst 
case regret (19) can be cast as an SDP problem. 
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Theorem 3: Let H, Hi , . . . , Hp e 
be full rank, then 



y, Yi, . . . , y G M™, 6a, 5b > 0, where m > n, and let H and H(q:) 



min max { ||H(a)u - y(/3)f - y^(I - HH+)y - 2Tr{V,^ ^, | 

is equivalent to solving the following SDP problem 

min 7 subject to 



^^y\\% /3]=o[" -3]}}' ^21) 



7 + e-T-0 (Hu-y)^ Sah^ 



Hu y 

(5bc 



I 

5G^ 
-SbQ" 



where ol 



A 



ai 02 



, H(a) = H + Er=i a^H„ /3 



(5G 
rl 


A 



Sbc'^ 

-SbQ 



ei 



> 0, (22) 



/3p ,y(/3)=y + Er=i/3^y^, 



b = Va||Piy(/3)f la=0,/3=0' c = V^||Pc!,y(/3)f la=0,/3=0' ^ = y^(I-HH+)y, r,^ > 0, G ^ [hiU H^u 

and Q - [yi ... yp . 

Proof: The outline of the proof follows similar lines to the proof of Theorem 1, hence it is omitted here. D 

In the following corollary, we provide a reduced version of Theorem 3, where the structure of the perturbations 
and the bounds on the perturbations are defined using the same parameters as commonly studied in [1], [2], i.e., if 
AH = Yl=i "^H„ then Ay = Yl=i "^y*' where H e M™^", y G K™ and a^ G R for alH = 1, . . . ,p. 
Corollary: Let H, Hi, . . . , Hp G M™^", y, yi, . . . , yp G K", 5 > 0, where ■m>n. Assume H and H(q:) are full 
rank, then minu maX||Q,||^j {|IH(q:)u - y(Q;)||^ - y^(I - HH+)y - 2Tr |VQ;||P^yp|^^Qa| } is equivalent 
to solving the following SDP problem 

min 7 subject to 



7 + C - T (Hu - y)^ 5h^ 



Hu y 

5h 



5G' 



5G 
tI 



>0, 



where a. = 



G 



a.\ Oi2 ... o.-^ 
Hiu yi Hau y2 



A 



H(a) = H + ELi "*H- y(«) = y + ELi ^^y^^ b = Va||Piy(a)||2|^^o, 
1 , ^ = y^(I - HH+)y, and t > 0. 



HpU 



HpU 



IV. Simulations 

We provide numerical examples in different scenarios in order to illustrate the merits of the proposed algorithms. 
In the first set of the experiments, we randomly generate a data matrix of size m x n, and an output vector of 
size m X 1, which are normalized to have unit norm. Then, we randomly generate 200 random perturbations AH, 
Ay, where ||AH|| < 6h, ||Ay[| < Sy, m = 5, n = 3, and Sh = Sy = 0.4. Here, we label the algorithm in 
Theorem 1 as "c-LS", the robust LS algorithm of [1] as "r-LS", and finally the LS algorithm tuned to the estimates 
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Fig. 1. Sorted errors for r-LS, c-LS and LS algorithms over Fig. 2. Averaged errors for r-LS, c-LS and LS algorithms 
200 trials when 5u = &y ^ 0.4. over 200 trials when 5 G [0.3, 06]. 

of the data matrix and the output vector as "L5"' where we solve (H -f AH)u = y + Ay. In Fig. 1, we plot 
the corresponding sorted errors in ascending order. Since the r-LS algorithm optimizes the worst case squared data 
error with respect to worst possible disturbance, it yields the smallest worst case squared error among all algorithms 
for these simulations. The largest errors are 1.585 for the LS algorithm, 1.267 for the c-LS algorithm and 1.120 
for the r-LS algorithm. Nevertheless, the overall performance of the r-LS algorithm is significantly inferior to the 
LS and the r-LS algorithms due to its highly conservative nature. Furthermore, we notice that the c-LS algorithm 
provides superior average performance compared to the LS and the r-LS algorithms, and superior worst case error 
compared to the LS algorithm for these simulations. Note that the LS algorithm yields the highest worst case eiTor. 
Although the worst case error of the c-LS algorithm is larger than the worst case error of the r-LS algorithm, the 
r-LS algorithm provides a superior performance on the average with respect to both the r-LS and the LS algorithms. 

For the second experiment, we randomly generate 200 random perturbations AH, Ay, where ||AH|| < 5//, 
ll^yll < 5y , m — ^, n — "i for different perturbation bounds and compute the averaged error over 200 trials for 
the c-LS, the LS, and the r-LS algorithms. In Fig. 2, we present the averaged LS eiTors for the c-LS, the LS, and 
the c-LS algorithms where the perturbation bound varies, by ^ ^h = ^ '^ [0.3, 0.6]. We observe that the proposed 
c-LS algorithm has the best average LS error performance over different perturbation bounds compared to the LS 
and r-LS algorithms. 

In the next experiment, we examine a system identification problem [2], which can be formulated as Uoh == yp, 
where U = Uq + W is the observed noisy Toeplitz matrix and y = yg + w is the observed noisy output vector. 
Here, the convolution matrix U (which is Toeplitz) constructed from u which is selected as a random sequence of 
±l's. For a randomly generated filter h of length 3, we generate 100 random structured perturbations for Uq and 
yg, where ||q:|| < 0.4||Uo||, and plotted the sorted errors in ascending order in Fig. 3. We observe that the largest 
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errors are 1.53 for the structured least squares bounded data uncertainties, labeled as "SLS-BDU" and presented in 
[2], 1.47 for the LS algorithm, 1.35 for the structured regret LS algorithm "sc-LS" from the Corollary, and 1.17 
for the stiTictured robust LS algorithm "sr-LS". We observe that the sr-LS algorithm yields the largest eiTor on 
the average, however, it yields the smallest worst case eiTor among other algorithms as expected. In addition, we 
observe that the sc-LS algorithm has a smaller worst case error compared to the LS and the SLS-BDU algorithms, 
and it has the smallest average error compared to the sr-LS, the SLS-BDU, and the LS algorithms, justifying its 
competitiveness in these simulations. Finally, in Fig. 4, we provide data errors sorted in ascending order for the 
algorithm in Theorem 2 as "c-RLS", for the robust regularized LS algorithm in [3] as "r-RLS" and finally for the 
regularized LS algorithm as "RLS" [4], where the experiment setup is the same as in the first experiment except 
the number of trials is 1000 and the perturbation bound is 0.3. The regularization parameter is chosen as /i = 0.1. 
In these simulations, we observe that the c-RLS algorithm trades off performance between the r-RLS and the RLS 
algorithms. The RLS algorithm yields the largest error compared to the r-RLS and c-RLS algorithms and we observe 
that although c-RLS has an inferior performance compared to the RLS algorithm on the average it yields a superior 
performance than the r-RLS algorithm. 

V. Conclusion 
In this correspondence, we introduced a novel robust approach to LS problems with bounded data uncertainties 
based on a certain regret formulation. We investigated the LS problems for both unstructured and structured 
perturbations, and the robust regularized LS problem for unstructured perturbations. In each case, the data vectors 
that minimize the worst case regrets are found by solving certain SDP problems. In our simulations, we observed 
that the proposed algorithms provide a fair trade off between performance and robustness, better than the best 
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available alternatives in different signal processing applications. 

Appendix 

Proof of Theorem 1: By applying 5-procedure [5] to (4), and with some algebra one can show that (4) is equivalent 
to solving the following SDP problem 

min7 subject to 

7 + C + c^h + h^c + Ay^b + b^Ay Chu - y J 
'hu y) I 



0< 



(23) 



Rearranging terms in (23) results 
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Here we used the identity AHu = Uh, where h = vec (AH). If we apply Proposition 2 of [7] into (24), we obtain 
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Applying Proposition 2 of [7] again in (25) yields (5). D 
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